Trainability and Accuracy of Artificial Neural Networks: An Interacting Particle System Approach

Neural networks, a central tool in machine learning, have demonstrated remarkable, high fidelity performance on image recognition and classification tasks. These successes evince an ability to accurately represent dimensional functions, but rigorous results about the approximation error of neural networks after training are few. Here we establish conditions for global convergence standard optimization algorithm used learning applications, stochastic gradient descent (SGD), quantify scaling its with size network. This is done by reinterpreting SGD as evolution particle system interactions governed potential related objective or "loss" function train We show that, when number $n$ units large, empirical distribution particles descends convex landscape towards minimum at rate independent $n$, resulting that universally scales $O(n^{-1})$. properties established form Law Large Numbers Central Limit Theorem distribution. Our analysis also quantifies scale nature noise introduced provides guidelines step batch use illustrate our findings examples which learn energy continuous 3-spin model sphere. The predicts dimension $d=25$.